Wave 1

For 1st wave, used an initial design of 200 points. Had emulators trained on this prior to the 3 hours, and just calculated implausibility once received actual observations.

Plotting ensemble and obs for the 7 flows:

Emulation

Glancing at some LOOs:

History matching

We don’t have any error, but need to set it at something. For now, just set as diagonal.

What does NROY look like?

Plotting the known runs that are in NROY:

Wave 2

Sampled a wave 2 restricted to runs we had available already. Plotting:

Refitting basis, emulators to new simulations. Reduced tolerance to error to force a smaller NROY space (size is arbitrary, only interested in `best’ runs):

Plotting W1+W2 distributions + overlaying truth:

Part of the perceived improvement at W2 could just be because reduced our error tolerance.

The 10 runs with lowest implausibility at wave 2:

##                f2         f3        fs2          fs3      alpha NROY_w2 NROY
## 1273  -0.09543336 -0.3480362  0.1631990  0.441243493 -0.3836307    TRUE TRUE
## 16182 -0.06373543 -0.3346185 -0.4355780  0.426106814 -0.3131207    TRUE TRUE
## 37699 -0.88673586 -0.3529626  0.7737754 -0.001525908 -0.2797586    TRUE TRUE
## 55417 -0.45261725 -0.3378206  0.8103676  0.191431676 -0.3764120    TRUE TRUE
## 61084 -0.01170803 -0.3497713  0.2576587  0.048704205 -0.2934673    TRUE TRUE
## 61158 -0.37902714 -0.3614865  0.9307290  0.204612706 -0.3064093    TRUE TRUE
## 68862 -0.58123997 -0.3593306  0.6799814 -0.016873887 -0.2807075    TRUE TRUE
## 73043 -0.08748638 -0.3269073 -0.7270260  0.583254196 -0.3643647    TRUE TRUE
## 76468  0.26808217 -0.3689451 -0.5316870  0.287363859 -0.3376002    TRUE TRUE
## 99134 -0.92063467 -0.3593677  0.9693147  0.171426453 -0.2742417    TRUE TRUE

Wave 3

Due to time constrains and lack of reliable access to Matlab or the internet (!!!), rather than sampling a new wave 3 and running ~100 new simulations, refitting emulators, and using these to get a single ‘best’ \(\textbf{x}^*\), instead stopped emulation here.

Took the 10 runs that minimised W2 implausibility, and ran these on the simulator.

Do these improve vs best run we’ve seen so far? How close can we get to the observations?

Plot the 10 new simulations:

Plotting these 10 runs on top of best run observed at waves 1 (blue) and 2 (purple):

Looking at true implausibility for 10 new runs, 4 are better than the best run we saw in W2:

##  [1] 1774.4139  340.2810 1006.4356 1949.9291  303.2708  404.7306  935.7751
##  [8]  565.2265  105.9921 1433.5852

Wave 4

Still had a bit of time, didn’t want to fit new emulators, so just perturbed the parameters around the current best run, and did 10 more simulations:

Looking at error for 10 new runs, 1 improves on best at W3:

##  [1]  397.32957  708.25180 1038.29911  280.64751   48.13425  392.97887
##  [7]  814.33111  481.11526  838.65244  330.32110

Adding the new best run to the earlier residual plot:

Just plotting W2/3/4 to see improvements:

Improvements?

What if we had emulated max/min pressure as well?

Look at residual vs true pressure are for the ‘best’ flow runs identified earlier (shown for W1-4):

## max_pressure min_pressure 
##    1.2940584    0.3943566

## max_pressure min_pressure 
##    0.1580628    0.5570256

##      V3585      V3586 
##  0.1830634 -0.2822653

##      V3585      V3586 
## -0.0669366  0.2347347

So in fact simply considering flows, we still improved our (average) fit to the max/min pressure each time.

Have we already found 1 of the closest runs from the 100k? 10 closest points in input space to the truth:

##  [1] 76468 62884  1098 37337 49477 29393 82240 11665 86491 45249

And in fact, sample 76468 was actually selected as one of the best 10 after wave 2, hence was simulated at wave 3; i.e. given we restricted to choosing from 100,000 inputs, we successfully identified the best possible choice (by emulating flow only, hence adding pressure cannot have improved these results).